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Abstract 


This research builds on the behavior learning theory that suggests a response from a student, followed 
by a quick feedback and another response from the student will increase student learning. An 
experiment was performed that allowed students to submit particular homework projects (response) 
early. The early submissions were graded promptly and returned to the students with comments for 
improvement (feedback). The students were then given the opportunity to resubmit the projects prior 
to the due date (another response) for final grading. Theory indicates that the students who took 
advantage of such a pre-grading option should do better on subsequent tests which would indicate 
increased learning as a result of the extra stimulus. The experimental results reported in the current 
paper provide partial support for the suggested increase in learning by those students who took 
advantage of the pre-grading option. 
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1. INTRODUCTION 

In many computer literacy courses, students are 
required to complete homework projects, in 
particular projects teaching how to use office 
productivity software, such as spreadsheets and 
databases. However, students often only 
receive a grade and some minor comments as 
feedback, and no option is given to correct the 
errors and actually learn from the mistakes. 
Learning theory suggests that increased learning 
will occur with additional stimuli and responses 
(Gagne, Briggs and Wager, 1992). Even though 


a number of studies in the research disciplines of 
pedagogy and psychology have addressed the 
argument of increased learning through added 
stimuli, the validity of this theory for the 
teaching of basic computer skills has not yet 
been established. In the current paper, we 
investigate the impact of pre¬ 
grading/resubmission of skill teaching projects 
on student learning. 

Over the past two years, we have used an 
automated grading tool in an introductory 
computer literacy course that is offered to 
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business students at a regional university. The 
use of the automated grading tool has 
significantly decreased the amount of time an 
instructor must dedicate to grading. 

The current paper emphasizes the capability of 
automated grading tools to support pre-grading 
of particular projects. First, a review of behavior 
learning theory is provided. It follows a review of 
automated grading tools and a description of the 
experiment with the statistical results. The paper 
concludes with a discussion of the results and 
potential extensions of the research. 

2. BACKGROUND: BEHAVIORAL LEARNING 
THEORY 

A student's success is influenced by the ability of 
the educator to present new information and to 
evaluate the student's understanding of the 
information. This process requires the student to 
learn the material covered by the educator. 

Based on the behavioral learning theory, Gagne 
et al. (1992) proposed several principles for the 
effective design of instructional courses, 
including contiguity, repetition, and feedback. 
Contiguity is the concept that the feedback 
should follow the response without delay. The 
longer the delay of the feedback to a learning 
stimulus the less is the likelihood of correct 
answers to future similar questions. The 
principle of repetition states that practice 
strengthens learning and improves a learner's 
retention. By combining the design principles 
Gagne et al. (1992) developed a conceptual 
framework of cognitive learning that includes 
nine "conditions for learning": 

• Gaining attention ("reception") 

• Informing learners of the objective 
("expectancy") 

• Stimulating recall of prior learning 
("retrieval") 

• Presenting the stimulus ("selective 
perception") 

• Providing learning guidance ("semantic 
encoding") 

• Eliciting performance ("responding") 

• Providing feedback ("reinforcement") 

• Assessing performance ("retrieval") 

• Enhancing retention and transfer 
(generalization") 

The results of subsequent research studies 
suggest that of the nine conditions, eliciting 
performance and practice from the student 
("responding") and providing adequate feedback 
("reinforcement") are the conditions most 


directly connected to student success (Martin, 
Klein & Sullivan, 2007). 

Similarly, Murray (1998) encouraged a teaching 
style based on drill/rote learning and 
memorization. Modules should be built with 
many exercises that are example driven. The 
principle of feedback requires that instructors 
inform the learner about whether the answer 
was correct or incorrect. In the case of an 
incorrect answer, feedback should include a new 
path to solve the problem. This new path could 
be a hint at the correct answer, a restatement of 
a prior fact, or even a new example that is less 
complicated (Uden & Beaumont, 2006). For 
instance, for an incorrect if-statement, a 
feedback explaining the binary nature of if- 
statements (true vs. false) would be 
appropriate. In addition, feedback that indicates 
that an answer is correct is just as important as 
feedback on incorrect answers. Confirming the 
correct answers would remove any doubt 
students might have on their newly learned skills 
and thus help retain the knowledge. 

Responding is required from learners after they 
have been given sufficient material to 
comprehend an objective (Tomei, 2008). A 
related process called orientation and recall is 
defined as a process where learning involves the 
synthesis of prior information that must be 
recalled to short term memory (Uden & 
Beaumont, 2006). Similarly, there is a school of 
thought that learners construct knowledge by 
making sense of experiences in terms of what is 
already known (Eugenia, 2010). 

When practice is included in a lesson, 
performance implies an active response by the 
student to the material provided. For example, 
in a database lesson, responding might require a 
student to create a query that counts the 
number of records in a table in order to 
demonstrate his/her comprehension of the newly 
introduced concept. 

Responding enables the student to reinforce 
his/her understanding. Effective practice should 
parallel the assessments that will be used to test 
skills and the knowledge reflected in an 
objective (Reiser & Dick, 1996). 

Building on on Gagne et al.'s (1992) results that 
response and reinforcement are key learning 
components, the current study investigates 
whether a focus on these key components can 
be helpful in teaching hands-on skills more 
effectively. 
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3. AUTOMATED GRADING TOOLS 

Automated grading systems are provided by a 
number of textbook publishers, among others. 
Key advantages of automated grading include: 

• Reduced lag time between submission of 
a project by a student, and response in 
the form of a grade or other feedback to 
the student 

• Application of a grading rubric for a 
project that is consistent for all students 

• Capability to add assignments as the 
grading time per project has been 
reduced. 

Indeed, the results of previous research studies 
suggest that the use of automated rubrics can 
result in faster and increased feedback, and that 
systems may be of advantage to instructors 
(Tan 2009; Anglin, Anglin, Schumman and 
Kalinski 2008; and Debuse, Lawley and Shibl 
2007). Similarly, Janicki and Steinberg (2003) 
suggested the need for increased computerized 
support for learning. Heinrich, Milne, Ramsay 
and Morrison (2009) demonstrated how e-tools 
can be used to increase the efficiency and 
quality of assignment making. 

Examples of automated grading systems include 
case-based auto graders and procedural-based 
graders. 

Case-based Auto Graders 

An example of a case-based auto grader is 
CASEGRADER by Thomson Course Technology 
(Crews and Murphy 2008). Instructors are 
provided with a set of cases that can be instantly 
graded. This type of system offers challenging, 
multi-step, realistic problems that students may 
submit to be automatically graded. Feedback is 
instantaneous and based on incorrect responses. 
Students are informed of their grade and 
provided feedback immediately following their 
submission of an assignment. One major 
limitation of this system includes the inability of 
instructors to create their own cases (Crews & 
Murphy, 2008). For example, for the Office 2007 
release, CASEGRADER offers a total of twelve 
different cases. If multiple sections of a course 
use the same limited set of cases, an increase in 
student plagiarism could occur. 

Procedural-based Graders 

Procedural-based graders include systems such 
as SAMS2007 (2007) by Thomson Course 
Technology and SNAP by EMC Paradigm 
Publishing (2007). These alternative systems are 


applications that grade student responses based 
on the procedure used to reach the answer. The 
application may either be a web system or a 
software application that simulates the 
environment of Microsoft Office programs in 
order to provide a hands-on experience for the 
students. These systems usually incorporate 
smaller problems that attempt to reinforce a 
procedure to be remembered. Few complex 
problems exist in the database of questions for 
these graders. 

4. PRE-GRADING WITH A CUSTOM-BUILT 
AUTOMATED GRADER 

Adaptive Grading and Learning System 

In order to meet the specific needs of students 
and instructors at a regional public university, a 
customized grader was developed and 
implemented in the fall of 2008. Known, as the 
Adaptive Grading/Learning System (AGLS), the 
system consists of modules that provide 
automated grading of Microsoft Excel 2007 and 
Access 2007 assignments with personalized and 
rapid feedback, assignment libraries that can be 
shared among participating instructors, and 
plagiarism detection. In addition, the system 
allows increasing the complexity of exercises 
without much additional effort by the instructor. 
This increase in complexity serves to challenge 
students and increases the likelihood of learning 
success. 

One result of the availability of the AGLS to 
instructors was a notable increase in the number 
of assignments that are given in class due to a 
significant decrease in grading time. For 
example, the instructor of one section of the 
computer literacy course now requires twelve 
different assignments, versus five projects that 
were required prior to the implementation of the 
AGLS four semesters ago. According to behavior 
learning theory, more responses from students 
should be associated with more learning. 

Pre-grading 

Following the introduction of the AGLS, some 
instructors gave students the opportunity to 
submit their projects in advance of the due date 
for one (or even several) round(s) of pre¬ 
grading. After a project was graded and specific 
comments were posted to the student's grade 
book on the web, the student could resubmit the 
project for final grading. 

It should be noted that the comments provided 
to the students did not give them the solution 
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but rather pointed to what needed to be 
corrected. Examples include: 

• Excel: Incorrect formula in B17 

• Excel: Missing IF in C24 

• Excel: Absolute reference in D22 

• Excel: Incorrect use of the SUM function 

• Access: Primary key incorrect in table 
'Customers' 

• Access: Field type incorrect for zip code 

• Access: Query Invoices, criteria for past 
due invalid 

The practice avoided students turning in a 
project basically blank and the automated 
system giving them the correct formulas or 
criteria. 

Methodology, Data Gathering and Analysis 

For the current study, experimental data was 
gathered from one section of eighty-seven 
students in an introductory information systems 
course. By selecting only one section taught by 
the same instructor the experiment avoided 
differences due to different instructor content, 
teaching styles and assignments. 


Assignment 

Number of student 
who submitted 
early 

Access (basic table 
and query design) 

76 out of 87 

Excel (basic IF's) 

66 out of 87 

Excel (Solver) 

61 out of 87 


Table 1: Pre-grading submissions 


Over one semester, students were given the 
opportunity to submit three assignments for pre¬ 
grading. Students only had to submit the 
assignment a reasonable time prior to the due 
date to get feedback and an opportunity to 
resubmit. Table 1 details the number of 
students who took advantage of the pre-grading 
opportunity. The order of the assignments in 
Table 1 is the order in which they were assigned 
during the semester. 

Two observations may be derived from Table 1. 
First, the number of students who submitted 
projects early was rather high. It was a very 
favorable observation that 85% of students 
submitted the first project early; and even at the 
end of the semester 70% of students submitted 
for pre-grading. The instructor expected the pre¬ 
grading rate to be lower. 


The second observation is less surprising: the 
number of students who submitted early 
decreased over the semester. The decrease can 
be interpreted such that as more work in other 
courses became due, students tended to 
complete their assignments closer to the due 
date. 

To test if additional learning occurred for those 
students who took advantage of the pre-grading 
opportunity, the following hypothesis was 
developed: 

Fl 0 - Pre-grading will not be associated 
with higher student scores 

FHj - Pre-grading will be associated with 
higher student scores 

In addition to the homework projects in Access 
and Excel (Table 1), four tests were 
administered during the semester. Each of the 
tests had two components: a multiple 
choice/short answer component and a hands-on 
component that tested the literacy skills covered 
in the previous weeks (i.e., Access and Excel). 
Pre-grading opportunities were available prior to 
three out of the four tests. 

For each test, student data was divided into two 
groups based on whether a student had taken 
advantage of the pre-grading option or not: 

• Experiment Group: took advantage of 
pre-grading prior to the test 

• Control Group: no pre-grading prior to 
the test 

For each test the population of the groups 
differed, based on who had taken advantage of 
the related pre-grading opportunity. Reflected in 
the numbers is, thus, the decreasing number of 
students who took advantage of pre-grading 
over the semester (Table 1). 

To eliminate any bias due to the differences of a 
student's prior knowledge or motivation, relative 
instead of absolute test scores were used. This 
measure also eliminates the potential that those 
students who submitted projects early were 
more motivated or more intelligent. Specifically, 
the difference between the scores of the multiple 
choice component and the hands-on component 
of the test for each student was used as the 
data-basis. For example: 

Experiment Group Student 1: 

Multiple Choice Test Score: 85 
Flands on Test Score 91 
Difference: 6 
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Control Group Student 1: 

Multiple Choice Test Score: 85 
Hands on Test Score 87 
Difference: 2 

Thus, in this example the experiment student 
scored 6 points higher on the hands-on 
component of the test while the control student 
scored 2 points higher. An analysis of the means 
was performed to determine whether the 
differences between the two groups were 
statistically significant. 

Results and Discussion 

For all cases a two tailed t-test was run 
assuming normality of the data. For two of the 
cases the variances test yielded unequal 
variances and thus a modified t-test was run 
(Table 2). 


Cases 

p-value 

Variance 

Access Hands On 

.114 

Unequal 

Excel IF's 

.524 

Unequal 

Excel Solver 

.012 

Equal 


Table 2: p values from t-tests 


As is summarized in Table 2, the results of the t- 
test analysis suggest that for two out of three 
assignments, the difference in learning as a 
result of pre-grading is NOT significant at the 
5%-level. For the third assignment (Excel 
Solver), however, a difference is found that is 
highly significant at 1.2%. In all, the support for 
learning theory as a result of our experiment 
appears to be mixed. 

A closer look at the data provides some 
additional insights and support for our 
hypothesis Hi. The first test (Access Hands-On) 
has a p-value of .114 which indicates acceptance 
of the null hypothesis; however, this result is 
close to a .10 p-value that can in fact be 
accepted for experimental research. One 
explanation for the highly insignificant t-test in 
the case of the second assignment/test may be 
that, while pre-grading concentrated on IF- 
statements, the test was actually on Excel 
Scenarios. IF-statements can be used in 
Scenarios, but are not necessarily included in 
the building of scenario cases that students 
often find difficult. Thus this hands-on test did 
not fully match the pre-grading assignment. 

Table 3 compares the average results of the 
control and experiment groups and computes 
the difference between the multiple choice and 
hands-on components for the entire section. The 
results support our H 1; as they suggest that 


there might in fact be a gain in learning from 
pre-grading: For the experiment group, the 
difference between the multiple-choice and the 
hands-on components of the test is larger (5.72) 
than for the control group (4.22). Students who 
took advantage of pre-grading performed 
particularly well in the hands-on component of 
the test when compared with the multiple-choice 
component. However, the difference was, again, 
not statistically significant the 5%-level. 



Control 

Experiment 

Avg Multiple 
Choice 

82.04 

86.06 

Avg Hands On 

86.27 

91.8 

Difference 

4.22 

5.72 


Table 3: Test score means and differences, both 
groups, all assignments combined 


An unanswered question is whether the scores 
on the multiple choice tests (which are higher 
for the experimental group) are not just a sign 
of more motivated students, but also a sign of 
learning of additional concepts from re-doing 
projects that then helped in answering questions 
on the multiple choice portion of the test. The 
effects of concept learning may thus have had 
an additional impact on the relative differences 
between the results of projects and tests and 
the levels of significance (or lack thereof). 

6. LIMITATIONS AND FUTURE RESEARCH 

One limitation of the current research setup 
resulted from the fact that the hands-on portion 
of the second test did not exactly match the 
concepts that were included in the pre-grading 
assignment. More specifically, pre-grading 
focused primarily on IF-statements while the 
test included scenario management skills in 
addition to IF-statements. In addition, there 
may have been an impact of concept learning 
from re-doing assignments. 

To overcome these limitations, the following 
research is currently in progress: 

A) Matching the concepts on the pre-graded 
assignments with the hands-on portions of 
the multiple choice tests; and 

B) A re-examination of the multiple choice tests 

to eliminate the impact of concept questions 
about Excel or Access on the final scores. 
This will permit a less biased analysis of the 
data. 
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7. SUMMARY AND CONCLUSIONS 

In summary, the results of the experiments 
partially supported the research hypothesis, and, 
thus provided limited support for the behavior 
learning theory that a response solicited from a 
student, followed by rapid feedback and then 
another response would increase student 
learning. Rapid feedback was assumed since all 
projects were graded within 48 hours of 
submission. In particular in the case in which the 
test concepts matched closely the hands-on 
concepts of the pre-grading project (Excel 
Solver), the results were highly significant, 
suggesting that learning did occur as a result of 
the pre-grading option. 

As a side-effect of the experiment, it was 
encouraging to see how many students took 
advantage of the pre-grading option, thus 
increasing their chance for learning. Another 
positive result of the pre-grading experiment 
was a noticeable reduction in the 'arguments' 
from students on grading. Since students were 
given the option to re-submit their projects, they 
did not argue over small grading questions. 
Whereas if the first grading had been final they 
might argue that =SUM(B3, B4, B5, B6) was a 
valid answer since it resulted in the correct value 
on the spreadsheet. The pre-grading option 
permits students to fix formulas that might have 
yielded the correct value, but were not 
considered the correct answer according to the 
learning objectives. 

The knowledge gained from this study provides 
valuable insights for instructors, particularly 
those teaching online web-based courses as 
such environment lacks the direct observation of 
learning during physical lab meetings. 
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